StepAudio 3 Gen is a new audio generation model that handles text-to-speech, voice design, sound effects, music, and speech all in one system using discrete autoregressive modeling instead of the diffusion approach most competitors use. It tokenizes audio at 12.5 Hz and generates by predicting codebooks sequentially, achieving state-of-the-art results on TTS and voice design tasks.

• Uses residual vector quantization tokens with a 16×2048 codebook space that jointly encodes semantic and waveform information, letting each layer preserve both types of data
• Employs a two-stage generation process: autoregressive prediction along the time axis for the first codebook, then a lightweight causal Transformer fills in the remaining 15 codebooks along the codebook axis
• Applies interference-aware progressive pretraining to add audio capabilities without degrading the underlying language model's text abilities